High Order LSTM/GRU

RNN is a powerful model for sequence data but suffers from gradient vanishing and explosion, thus difficult to be trained to capture long range dependences. But people have proposed LSTM and GRU, which try to model the differences between adjacent data frame rather than the data frame itself. By doing so, it allows the error to back propagate throw longer time without vanishing. Also instead of simply accumulating information, these models introduce multiple gating functions which depend on current input and hidden states to control whether information should be taken-in, passed though or discarded. Currently all the gating functions are sigmoid/tanh function over a linear combination of input x and current hidden states h. In this work, we are trying to introduce a high order term for RNN as a component of gating function. By doing so, we argue that it can better model the relation between current input and hidden states, thus better control the flow of information and learn better representation accordingly.